Methods for optimized variable-size deduplication using two stage content-defined chunking and devices thereof

ABSTRACT

Methods, non-transitory machine readable media, and computing devices that compare a hash value to a predefined value for sliding windows in parallel for segments partitioned from an input data stream. A bit array is parsed according to minimum and maximum chunk sizes to identify chunk boundaries for the input data stream. The bit array is populated based on a result of the comparison and portions of the bit array are parsed in parallel. Unique chunks of the input data stream defined by the chunk boundaries are stored in a storage device. Accordingly, this technology utilizes parallel processing in two stages. In a first stage, rolling window based hashing is performed concurrently to identify potential chunk boundaries. In a second stage, actual chunk boundaries are selected based on minimum and maximum chunk size constraints. This technology advantageously facilitates significant deduplication ratio improvement as well as improved parallel chunking performance.

This application claims the benefit of U.S. Provisional PatentApplication Ser. No. 62/743,352, filed Oct. 9, 2018, which is herebyincorporated by reference in its entirety.

FIELD

This technology generally relates to deduplication in data storagenetworks and, more particularly, to methods and devices for optimizingvariable-size deduplication using two stage content-defined chunking.

BACKGROUND

Data deduplication, a technique for data reduction, has been widelydeployed in storage systems to improve storage efficiency and I/Operformance. In chunk-based deduplication, the input data stream ispartitioned into chunks, and only unique chunks are stored.

There are two approaches to chunk-based deduplication: fixed-sizechunking and variable-size chunking. In fixed-size chunking, the inputdata stream is partitioned into fixed-size chunks. Fixed-size chunkingis relatively fast because it does not require any computation based oninput data. However, insertions and deletions to the same file willintroduce boundary shifts, leading to a relatively poor deduplicationratio.

To solve this problem, variable-size chunking was developed and thenfurther improved through the use of content-defined variable-sizechunking. In content-defined variable-size chunking, the chunk boundaryis determined by the content of the input data stream. In a typicalcontent-defined chunking implementation, a fixed-size sliding window isused and, for each sliding window, a hash value is determined over thecontents and compared with a pre-defined value.

If the hash value matches the pre-defined value, the end of the windowis declared as a chunk boundary. Otherwise, the sliding window is movedforward and the hash computation and comparison process is repeated forthe new sliding window. When minimum and maximum chunk sizes arespecified, the sliding window hash process is started after skipping thefirst minimum chunk size bytes. A chunk boundary is then declared whenthe maximum chunk size is reached and no chunk boundary is yet detectedfrom the sliding window hash process.

However, the sliding-window based hash process is very computeintensive, as it requires a hash calculation for every sliding window.Thus, if performed sequentially, the performance will be relativelypoor. To improve performance, the input data can be partitioned intoequal-size segments and the content-based chunking can be performed inparallel using parallel hardware, which is referred to as parallelcontent-defined chunking.

Since each segment is processed independently at each processing unit inparallel content-defined chunking, a chunk is forced to cut at the endof each segment. In particular, the last chunk that is forced to cutfrom the previous segment may span over to the current segment. Spanningto the current segment can lead to different starting points since thefirst minimum chunk size is always skipped. Accordingly, compared tosequential content-defined chunking, the segment-based deduplication isless efficient in detecting duplicate blocks, leading to significantdegradation in deduplication ratio.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a network environment with exemplary nodecomputing devices;

FIG. 2 is a block diagram of one of the exemplary node computing deviceof FIG. 1;

FIG. 3 is a flowchart of an exemplary method for optimized variable-sizededuplication using two stage content-defined chunking;

FIG. 4 is a flow diagram illustrating exemplary two stage parallelcontent-defined chunking;

FIG. 5 is a flow diagram illustrating exemplary parallel chunk boundarydetermination;

FIG. 6 is a set of graphs illustrating deduplication ratio improvementsof examples of this technology over parallel content-defined chunking(CDC) with variable average chunk size and fixed minimum and maximumchunk sizes;

FIG. 7 is a set of graphs illustrating deduplication ratio improvementsof examples of this technology over parallel CDC with variable minimumchunk sizes; and

FIG. 8 is a set of graphs illustrating improved chunking speed ofexamples of this technology over sequential CDC with variable minimumchunk size.

DETAILED DESCRIPTION

A clustered network environment 100 that may implement one or moreaspects of the technology described and illustrated herein is shown inFIG. 1. The clustered network environment 100 includes data storageapparatuses 102(1)-102(n) that are coupled over a cluster fabric 104facilitating communication between the data storage apparatuses102(1)-102(n) (and one or more modules, components, etc. therein, suchas, node computing devices 106(1)-106(n), for example), although anynumber of other elements or components can also be included in theclustered network environment 100 in other examples. This technologyprovides a number of advantages including methods, non-transitorycomputer readable media, and computing devices that facilitate improveddeduplication ratios and performance.

In this example, node computing devices 106(1)-106(n) can be primary orlocal storage controllers or secondary or remote storage controllersthat provide client devices 108(1)-108(n) with access to data storedwithin data storage devices 110(1)-110(n). The data storage apparatuses102(1)-102(n) and/or node computing devices 106(1)-106(n) of theexamples described and illustrated herein are not limited to anyparticular geographic areas and can be clustered locally and/orremotely, or not clustered in other examples. Thus, in one example thedata storage apparatuses 102(1)-102(n) and/or node computing device106(1)-106(n) can be distributed over a plurality of storage systemslocated in a plurality of geographic locations; while in another examplea clustered network can include data storage apparatuses 102(1)-102(n)and/or node computing device 106(1)-106(n) residing in a same geographiclocation (e.g., in a single onsite rack).

In the illustrated example, one or more of the client devices108(1)-108(n), which may be, for example, personal computers (PCs),computing devices used for storage (e.g., storage servers), or othercomputers or peripheral devices, are coupled to the respective datastorage apparatuses 102(1)-102(n) by network connections 112(1)-112(n).Network connections 112(1)-112(n) may include a local area network (LAN)or wide area network (WAN), for example, that utilize Network AttachedStorage (NAS) protocols, such as a Common Internet Filesystem (CIFS)protocol or a Network Filesystem (NFS) protocol to exchange datapackets, a Storage Area Network (SAN) protocol, such as Small ComputerSystem Interface (SCSI) or Fiber Channel Protocol (FCP), an objectprotocol, such as simple storage service (S3), and/or non-volatilememory express (NVMe), for example.

Illustratively, the client devices 108(1)-108(n) may be general-purposecomputers running applications and may interact with the data storageapparatuses 102(1)-102(n) using a client/server model for exchange ofinformation. That is, the client devices 108(1)-108(n) may request datafrom the data storage apparatuses 102(1)-102(n) (e.g., data on one ofthe data storage devices 110(1)-110(n) managed by a network storagecontroller configured to process I/O commands issued by the clientdevices 108(1)-108(n)), and the data storage apparatuses 102(1)-102(n)may return results of the request to the client devices 108(1)-108(n)via the network connections 112(1)-112(n).

The node computing devices 106(1)-106(n) of the data storage apparatuses102(1)-102(n) can include network or host nodes that are interconnectedas a cluster to provide data storage and management services, such as toan enterprise having remote locations, cloud storage (e.g., a storageendpoint may be stored within a data cloud), etc., for example. Suchnode computing devices 106(1)-106(n) can be attached to the fabric 104at a connection point, redistribution point, or communication endpoint,for example. One or more of the node computing devices 106(1)-106(n) maybe capable of sending, receiving, and/or forwarding information over anetwork communications channel, and could comprise any type of devicethat meets any or all of these criteria.

In an example, the node computing devices 106(1) and 106(n) may beconfigured according to a disaster recovery configuration whereby asurviving node provides switchover access to the storage devices110(1)-110(n) in the event a disaster occurs at a disaster storage site(e.g., the node computing device 106(1) provides client device 112(n)with switchover data access to storage devices 110(n) in the event adisaster occurs at the second storage site). In other examples, the nodecomputing device 106(n) can be configured according to an archivalconfiguration and/or the node computing devices 106(1)-106(n) can beconfigured based on another type of replication arrangement (e.g., tofacilitate load sharing). Additionally, while two node computing devicesare illustrated in FIG. 1, any number of node computing devices or datastorage apparatuses can be included in other examples in other types ofconfigurations or arrangements.

As illustrated in the clustered network environment 100, node computingdevices 106(1)-106(n) can include various functional components thatcoordinate to provide a distributed storage architecture. For example,the node computing devices 106(1)-106(n) can include network modules114(1)-114(n) and disk modules 116(1)-116(n). Network modules114(1)-114(n) can be configured to allow the node computing devices106(1)-106(n) (e.g., network storage controllers) to connect with clientdevices 108(1)-108(n) over the storage network connections112(1)-112(n), for example, allowing the client devices 108(1)-108(n) toaccess data stored in the clustered network environment 100.

Further, the network modules 114(1)-114(n) can provide connections withone or more other components through the cluster fabric 104. Forexample, the network module 114(1) of node computing device 106(1) canaccess the data storage device 110(n) by sending a request via thecluster fabric 104 through the disk module 116(n) of node computingdevice 106(n). The cluster fabric 104 can include one or more localand/or wide area computing networks embodied as Infiniband, FibreChannel (FC), or Ethernet networks, for example, although other types ofnetworks supporting other protocols can also be used.

Disk modules 116(1)-116(n) can be configured to connect data storagedevices 110(1)-110(2), such as disks or arrays of disks, SSDs, flashmemory, or some other form of data storage, to the node computingdevices 106(1)-106(n). Often, disk modules 116(1)-116(n) communicatewith the data storage devices 110(1)-110(n) according to the SANprotocol, such as SCSI or FCP, for example, although other protocols canalso be used. Thus, as seen from an operating system on node computingdevices 106(1)-106(n), the data storage devices 110(1)-110(n) can appearas locally attached. In this manner, different node computing devices106(1)-106(n), etc. may access data blocks, files, or objects throughthe operating system, rather than expressly requesting abstract files.

While the clustered network environment 100 illustrates an equal numberof network modules 114(1)-114(2) and disk modules 116(1)-116(n), otherexamples may include a differing number of these modules. For example,there may be a plurality of network and disk modules interconnected in acluster that do not have a one-to-one correspondence between the networkand disk modules. That is, different node computing devices can have adifferent number of network and disk modules, and the same nodecomputing device can have a different number of network modules thandisk modules.

Further, one or more of the client devices 108(1)-108(n) can benetworked with the node computing devices 106(1)-106(n) in the cluster,over the storage connections 112(1)-112(n). As an example, respectiveclient devices 108(1)-108(n) that are networked to a cluster may requestservices (e.g., exchanging of information in the form of data packets)of node computing devices 106(1)-106(n) in the cluster, and the nodecomputing devices 106(1)-106(n) can return results of the requestedservices to the client devices 108(1)-108(n). In one example, the clientdevices 108(1)-108(n) can exchange information with the network modules114(1)-114(n) residing in the node computing devices 106(1)-106(n)(e.g., network hosts) in the data storage apparatuses 102(1)-102(n).

In one example, the storage apparatuses 102(1)-102(n) host aggregatescorresponding to physical local and remote data storage devices, such aslocal flash or disk storage in the data storage devices 110(1)-110(n),for example. One or more of the data storage devices 110(1)-110(n) caninclude mass storage devices, such as disks of a disk array. The disksmay comprise any type of mass storage devices, including but not limitedto magnetic disk drives, flash memory, and any other similar mediaadapted to store information, including, for example, data and/or parityinformation.

The aggregates include volumes 118(1)-118(n) in this example, althoughany number of volumes can be included in the aggregates. The volumes118(1)-118(n) are virtual data stores or storage objects that define anarrangement of storage and one or more filesystems within the clusterednetwork environment 100. Volumes 118(1)-118(n) can span a portion of adisk or other storage device, a collection of disks, or portions ofdisks, for example, and typically define an overall logical arrangementof data storage. In one example volumes 118(1)-118(n) can include storeduser data as one or more files, blocks, or objects that reside in ahierarchical directory structure within the volumes 118(1)-118(n).Volumes 118(1)-118(n) are typically configured in formats that may beassociated with particular storage systems, and respective volumeformats typically comprise features that provide functionality to thevolumes 118(1)-118(n), such as providing the ability for volumes118(1)-118(n) to form clusters, among other functionality.

In one example, to facilitate access to data stored on the disks orother structures of the data storage devices 110(1)-110(n), a filesystemmay be implemented that logically organizes the information as ahierarchical structure of directories and files. In this example,respective files may be implemented as a set of disk blocks of aparticular size that are configured to store information, whereasdirectories may be implemented as specially formatted files in whichinformation about other files and directories are stored.

Data can be stored as files or objects within a physical volume and/or avirtual volume, which can be associated with respective volumeidentifiers. The physical volumes correspond to at least a portion ofphysical storage devices, such as the data storage devices 110(1)-110(n)(e.g., a Redundant Array of Independent (or Inexpensive) Disks (RAIDsystem)) whose address, addressable space, location, etc. does notchange. Typically the location of the physical volumes does not changein that the range of addresses used to access it generally remainsconstant.

Virtual volumes, in contrast, can be stored over an aggregate ofdisparate portions of different physical storage devices. Virtualvolumes may be a collection of different available portions of differentphysical storage device locations, such as some available space fromdisks, for example. It will be appreciated that since the virtualvolumes are not “tied” to any one particular storage device, virtualvolumes can be said to include a layer of abstraction or virtualization,which allows it to be resized and/or flexible in some regards.

Further, virtual volumes can include one or more logical unit numbers(LUNs), directories, Qtrees, files, and/or other storage objects, forexample. Among other things, these features, but more particularly theLUNs, allow the disparate memory locations within which data is storedto be identified, for example, and grouped as data storage unit. Assuch, the LUNs may be characterized as constituting a virtual disk ordrive upon which data within the virtual volumes is stored within anaggregate. For example, LUNs are often referred to as virtual drives,such that they emulate a hard drive, while they actually comprise datablocks stored in various parts of a volume.

In one example, the data storage devices 110(1)-110(n) can have one ormore physical ports, wherein each physical port can be assigned a targetaddress (e.g., SCSI target address). To represent respective volumes, atarget address on the data storage devices 110(1)-110(n) can be used toidentify one or more of the LUNs. Thus, for example, when one of thenode computing devices 106(1)-106(n) connects to a volume, a connectionbetween the one of the node computing devices 106(1)-106(n) and one ormore of the LUNs underlying the volume is created.

Respective target addresses can identify multiple of the LUNs, such thata target address can represent multiple volumes. The I/O interface,which can be implemented as circuitry and/or software in a storageadapter or as executable code residing in memory and executed by aprocessor, for example, can connect to volumes by using one or moreaddresses that identify the one or more of the LUNs.

Referring to FIG. 2, node computing device 106(1) in this particularexample includes processor(s) 200, a memory 202, a network adapter 204,a cluster access adapter 206, and a storage adapter 208 interconnectedby a system bus 210. The node computing device 106 also includes astorage operating system 212 installed in the memory 206 that can, forexample, implement a RAID data loss protection and recovery scheme tooptimize reconstruction of data of a failed disk or drive in an array.In some examples, the node computing device 106(n) is substantially thesame in structure and/or operation as node computing device 106(1),although the node computing device 106(n) can also include a differentstructure and/or operation in one or more aspects than the nodecomputing device 106(1).

The network adapter 204 in this example includes the mechanical,electrical and signaling circuitry needed to connect the node computingdevice 106(1) to one or more of the client devices 108(1)-108(n) overnetwork connections 112(1)-112(n), which may comprise, among otherthings, a point-to-point connection or a shared medium, such as a localarea network. In some examples, the network adapter 204 furthercommunicates (e.g., using TCP/IP) via the fabric 104 and/or anothernetwork (e.g. a WAN) (not shown) with cloud storage devices to processstorage operations associated with data stored thereon.

The storage adapter 208 cooperates with the storage operating system 212executing on the node computing device 106(1) to access informationrequested by one of the client devices 108(1)-108(n) (e.g., to accessdata on a data storage device 110(1)-110(n) managed by a network storagecontroller). The information may be stored on any type of attached arrayof writeable media such as magnetic disk drives, flash memory, and/orany other similar media adapted to store information.

In the exemplary data storage devices 110(1)-110(n), information can bestored in data blocks on disks. The storage adapter 208 can include I/Ointerface circuitry that couples to the disks over an I/O interconnectarrangement, such as a storage area network (SAN) protocol (e.g., SmallComputer System Interface (SCSI), Internet SCSI (iSCSI), hyperSCSI,Fiber Channel Protocol (FCP)). The information is retrieved by thestorage adapter 208 and, if necessary, processed by the processor(s) 200(or the storage adapter 208 itself) prior to being forwarded over thesystem bus 210 to the network adapter 204 (and/or the cluster accessadapter 206 if sending to another node computing device in the cluster)where the information is formatted into a data packet and returned to arequesting one of the client devices 108(1)-108(2) and/or sent toanother node computing device attached via the cluster fabric 104. Insome examples, a storage driver 214 in the memory 202 interfaces withthe storage adapter to facilitate interactions with the data storagedevices 110(1)-110(n).

The storage operating system 212 can also manage communications for thenode computing device 106(1) among other devices that may be in aclustered network, such as attached to a cluster fabric 104. Thus, thenode computing device 106(1) can respond to client device requests tomanage data on one of the data storage devices 110(1)-110(n) (e.g., oradditional clustered devices) in accordance with the client devicerequests.

The file system module 216 of the storage operating system 212 canestablish and manage one or more filesystems including software code anddata structures that implement a persistent hierarchical namespace offiles and directories, for example. As an example, when a new datastorage device (not shown) is added to a clustered network system, thefile system module 216 is informed where, in an existing directory tree,new files associated with the new data storage device are to be stored.This is often referred to as “mounting” a filesystem.

In the example node computing device 106(1), memory 202 can includestorage locations that are addressable by the processor(s) 200 andadapters 204, 206, and 208 for storing related software application codeand data structures. The processor(s) 200 and adapters 204, 206, and 208may, for example, include processing elements and/or logic circuitryconfigured to execute the software code and manipulate the datastructures. The processor(s) in some examples comprise parallelprocessing and, in particular, can support Advanced Vector Extensions(AVX) extensions to the x86 instruction set architecture, as describedand illustrated in more detail later.

The storage operating system 212, portions of which are typicallyresident in the memory 202 and executed by the processor(s) 200, invokesstorage operations in support of a file service implemented by the nodecomputing device 106(1). Other processing and memory mechanisms,including various computer readable media, may be used for storingand/or executing application instructions pertaining to the techniquesdescribed and illustrated herein. For example, the storage operatingsystem 212 can also utilize one or more control files (not shown) to aidin the provisioning of virtual machines.

In this particular example, the storage operating system 212 furtherincludes a deduplication module 218 that is configured to reduce thestorage space utilized on one or more of the data storage devices110(1)-110(n). The deduplication module 218 processes input data streamsto facilitate identification, and storing on one or more of the datastorage devices 110(1)-110(n), of unique chunks of the input data streamin order to improve storage utilization and optimize storageperformance, for example, as described and illustrated in more detaillater with reference to FIGS. 3-8.

The examples of the technology described and illustrated herein may beembodied as one or more non-transitory computer readable media havingmachine or processor-executable instructions stored thereon for one ormore aspects of the present technology, which when executed by theprocessor(s) 200, cause the processor(s) 200 to carry out the stepsnecessary to implement the methods of this technology, as described andillustrated with the examples herein. In some examples, the executableinstructions are configured to perform one or more steps of a method,such as one or more of the exemplary methods described and illustratedlater with reference to FIGS. 3-6, for example.

Referring more specifically to FIG. 3, a flow diagram illustrating anexemplary method for optimized variable-size deduplication using twostage content-defined chunking is illustrated. In step 300 in thisexample, the node computing device 106(1) partitions an input datastream into segments of equal size (e.g., 1 MB, although other segmentsizes can also be used. The input data stream can be obtained from oneof the storage devices 110(1)-110(n) as part of a backgrounddeduplication operation or can be received from one of the clientdevices 108(1)-108(n) as part of an initial storage operation, althoughother types of methods for obtaining the input data stream can be usedin other examples. While the examples are described and illustratedherein with reference to the node computing device 106(1), theseexamples can be implemented by any one or more of the node computingdevices 106(1)-106(n).

In step 302, the node computing device 106(1) compares a hash value to apredefined value for sliding windows in parallel for each of thesegments partitioned from the input data stream. The node computingdevice 106(1) uses a sliding window starting at the beginning of eachsegment to calculate a hash value and compare the hash value to apredefined value. In this example, each segment is assigned to one of aplurality of threads executing in parallel. Accordingly, the nodecomputing device 106(1) assigns each thread one of the segments and therolling or sliding window hashing and comparison is performedindependently and in parallel by each thread.

In this particular example, one or more of the sliding windows for atleast a subset of the segments extends into a contiguous or next one ofthe segments. Accordingly, in each thread, the sliding window will rollover to an additional portion from the beginning of the next segment,such that a sliding window covers every portion (e.g., byte) in thecurrent segment.

The node computing device 106(1) in this example outputs a 0 or 1 foreach sliding window, although other numbers or values can be used. Inthis particular example, 0 indicates that the hash value does not matchthe predefined value and the associated sliding window is not acandidate as a chunk boundary, whereas 1 indicates that the hash valuematches the predefined value and the sliding window is a candidate as achunk boundary.

In step 304, the node computing device 106(1) stores a result of thecomparisons in a bit array data structure, although other types of datastructures can be used in other examples. In particular, the nodecomputing device 106(1) sets one of the bits in the bit array when thehash value matches the predefined value for one of the sliding windows.In other words, the node computing device 106(1) sets an n^(th) bit inthe bit array when the hash value matches the predefined value at anoffset of n in the input data stream. The set one of the bits representsa chunk boundary candidate.

Accordingly, each of the bits of the bit array corresponds to one of thesliding windows and records whether a match was found of the hash valueat the associated offset. In some examples, for an input data streamwith N bytes, the output bit array will be N bits. Following step 304 inthis example, the node computing device 106(1) will have constructed anarray of bits with values of 0s and 1s and each value corresponding toone sliding window in the original input data stream and representingwhether the associated sliding window corresponds with a candidate chunkboundary.

Referring more specifically to FIG. 4, a flow diagram illustratingexemplary two stage parallel content-defined chunking is illustrated. Inthis example, the node computing device 106(1) partitions the input datastream 400 into equal size segments 402(1), 402(2), 402(3), and 402(4).Each of the segments 402(1), 402(2), 402(3), and 402(4) is thendispatched to a thread for detecting potential or candidate chunkboundaries. Accordingly, the node computing device 106(1) in thisexample executes four threads, one for each of segments 402(1), 402(2),402(3), and 402(4). The threads will continue the rolling or slidingwindow hashing for portions 404(1), 404(2), and 404(3) of the next orcontinuous segments.

Accordingly, the rolling window hashing is performed to identify allpotential or candidate chunk boundaries and no final chunk boundariesare determined at this stage. Additionally, each thread not only checksthe regular segments (i.e., segments 402(1), 402(2), 402(3), and402(4)), but also checks a portion or set of bytes (i.e., portions404(1), 404(2), and 404(3) in this example) from the next segment (i.e.,segments 402(2), 402(3), and 402(4), respectively, in this example),thereby ensuring that a hash value is generated for every portion orbyte in the current segment.

Based on whether there is a match of the hash value with the predefinedvalue for each of the sliding windows, a bit array 406 is populated bythe node computing device 106(1). Accordingly, the bit array 406includes a 1 value in this example for each sliding window for which thegenerated hash value matched the predefined value and a 0 value for eachsliding window for which the generated hash value did not match thepredefined value. The 1s in the bit array 406 represent candidate chunkboundaries for the input data stream. The bit array 406 is provided asinput to a second stage in which final chunk boundaries are determinedfrom the candidate chunk boundaries, as described and illustrated inmore detail with reference to steps 306-324 of FIG. 3 and in FIG. 5.

Referring specifically to FIG. 3, in step 306, the node computing device106(1) generates a set of integers from a subset of the bitscorresponding to a portion of the bit array generated in step 304. Thesubset of the bits used to generate the set of integers follows aminimum chunk size, which can be configured and established by policyand/or stored in the memory 202, for example. Skipping the minimum chunksize when generating the set of integers effectively allows the nodecomputing device 106(1) to ignore any candidate chunk boundaries thatmay be represented in the bit array within the bits corresponding to theminimum chunk size.

In step 308, the node computing device 106(1) analyzes the set ofintegers using parallel hardware to determine whether any of the set ofintegers includes a candidate chunk boundary. In this particularexample, the hardware includes advanced vector extension (AVX)register(s) and processor(s) 200 configured to execute AVX instructions(e.g., AVX-512 instructions), which are extensions to the x86instruction set architecture for microprocessors that enable performanceof vector operations in a single instruction, multiple data (SIMD)manner. For processors with AVX-512 instructions support, as oneexample, the extended registers are 512-bit long. The use of IntelAVX-512 instructions can enable a 512-bit, or 16 32 bit integer, checkwith one instruction.

The check in this example is to determine whether any of the set ofintegers (e.g., 16 integers) includes a 1 that represents a candidatechunk boundary. Accordingly, the node computing device 106(1) loads theset of integers into an AVX register and one of the processor(s) 200works as a SIMD processor with 16 concurrent threads in this example,each operating on a 32-bit value, to generate an indication of whetherany of the integers includes a 1 value representing a candidate chunkboundary. The formation and processing of the set of integers allows thenode computing device 106(1) to effectively skip the 0s in the bitarray, which will be the most common occurring value, and identify finalchunk boundaries more efficiently.

In step 310, the node computing device 106(1) determines whether thereis a candidate chunk boundary in the analyzed set of integers generatedfrom the bit array based on the processing of the analyzed set ofintegers by the parallel hardware. If the node computing device 106(1)does not determine that there is a candidate chunk boundary in the setof integers, then the No branch is taken back to step 306, and the nodecomputing device 106(1) again generates another set of integers frombits corresponding to a portion of the bit array.

Accordingly, when the No branch is taken from step 310, the set ofintegers comprised all 0 values in this example, facilitating relativelyefficient processing of the bit array (e.g., as compared to a sequentialscan of the bit array). However, if the node computing device 106(1)determines that there is a candidate chunk boundary in the set ofintegers (i.e. a 1 value in at least one of the bits from the bit arrayfrom which at least one of the integers in the set was generated), thenthe Yes branch is taken to step 312.

In step 312, the node computing device 106(1) sequentially scans the setof integers. In this example, the node computing device 106(1) scans theset of integers in the AVX register to identify a first one of the setof integers that includes a 1 value representing a candidate chunkboundary.

Referring to FIG. 5, a flow diagram illustrating exemplary parallelchunk boundary determination is illustrated. In this example, afterskipping the minimum chunk size of bits, every 512 bits are loaded bythe node computing device 106(1) from a bit array (e.g., bit array 406)as 16 32-bit integers into an AVX-512 register. Then, an AVX instruction(e.g., _mm512_cmpneq_epi32_mask) is executed by the node computingdevice 106(1) to determine whether all 16 integers comprise 0s. If anyof these 16 32 bit integers is not equal to 0, then a sequential scan isperformed by the node computing device 106(1) within the 16 integers tofind the first integer which is non-zero.

Referring back to FIG. 3, in step 314, the node computing device 106(1),while scanning the set of integers, determines whether a first one ofthe set of integers that includes the candidate chunk boundary isidentified. If the node computing device 106(1) does not determine thatthe first one of the set of integers that includes the candidate chunkboundary is identified, then the No branch is taken to step 316.

In step 316, the node computing device 106(1) determines whether amaximum chunk size has been reached. The maximum chunk size can beconfigured and established by a policy and/or may be stored in thememory 202, for example. In one example, the node computing device106(1) maintains a current chunk size based on the identification of alast final chunk boundary from one of the candidate chunk boundaries, asdescribed and illustrated in more detail below.

For example, if the node computing device 106(1) determines in step 310that a set of integers does not include a candidate chunk boundary, thenthe current chunk size can be incremented based on a size correspondingto the number of sliding windows in a segment of the input data streamrepresented by the bits in the set of integers. In another example inwhich the maximum chunk size may be relatively small (e.g.,corresponding to less than a set of integers), the current chunk sizecan be incremented as the node computing device 106(1) sequentiallyscans a set of integers in steps 312-314 to determine whether enough 0bits have been encountered such that the maximum chunk size has beenreached.

If the current chunk size reaches the maximum chunk size, then the nodecomputing device 106(1) establishes a final chunk boundary irrespectiveof whether a first one of the set of integers including a candidatechunk boundary has been reached as part of the sequential scan. If thenode computing device 106(1) determines in step 316 that the maximumchunk size has not been reached, then the No branch is taken back tostep 312, and the node computing device 106(1) continues sequentiallyscanning the set of integers.

Accordingly, the node computing device 106(1) effectively sequentiallyscans the set of integers until a first one of the set of integers thatincludes a candidate boundary is identified or the maximum chunk size isreached. Referring back to step 314, if the node computing device 106(1)determines that a first one of the set of integers is identified thatincludes a candidate chunk boundary, then the Yes branch is taken tostep 318.

In step 318, the node computing device 106(1) left-shifts the first oneof the set of integers until the left-most bit comprises a value(e.g., 1) that corresponds to the candidate chunk boundary. Since thecandidate boundary is beyond a minimum chunk size from the prior chunk,if any, and within the maximum chunk size, the candidate boundary nowrepresents a final chunk boundary. Subsequent to left-shifting the firstone of the set of integers until the left-most bit corresponds to thecandidate chunk boundary in step 318, or if the node computing device106(1) determines in step 316 that the maximum chunk size is reached andthe Yes branch is taken, then the node computing device 106(1) proceedsto step 320.

In step 320, the node computing device 106(1) creates a chunk of theinput data stream. If the first one of the set of integers wasidentified in step 314 prior to the maximum chunk size being reached,then the chunk of the input data stream is created based on thecandidate chunk boundary, which is not a final chunk boundary. However,if the maximum chunk size was reached in step 316 before the first oneof the set of integers was identified, then the chunk of the input datastream is created based on the bit of one of the set of integers thatwas reached during the sequential scan upon satisfaction of the maximumchunk size.

The bit corresponding to the final chunk boundary or the maximum chunksize is at an offset in the set of integers that corresponds with alocation of one of the sliding window for one of the segmentspartitioned from the input data stream. Accordingly, the chunk of theinput data stream is created at a portion of the sliding window (e.g.,the beginning) within the one of the partitioned segments.

In step 322, the node computing device 106(1) determines whether the endof the bit array has been reached. If the node computing device 106(1)determines that the end of the bit array has not been reached, then theNo branch is taken back to step 306, and another set of integers isgenerated. Accordingly, the node computing device 106(1) effectivelyrepeats steps 306-322 for the remainder of the bit array until all ofthe final chunk boundaries are identified. If the node computing device106(1) determines that the end of the bit array has been reached, thenthe Yes branch is taken to step 324.

In step 324, the node computing device 106(1) stores unique ones of thecreated chunks of the input data stream that are defined by the finalchunk boundaries in a storage device, such as one of data storage devise110(1)-110(n) for example. Subsequent to storing the unique ones of thecreated chunks, the node computing device 106(1) proceeds back to step300 and partitions another input data stream in this example. In otherexamples, one or more of steps 300-324 can be performed in a differentorder and/or in parallel.

Referring to FIGS. 6-8 various testing results for this technology areillustrated. The tests were conducted on an EC2 C5 large instance serverwith two virtual central processing units (vCPUs) provided by Amazon WebServices™. The server processors supported Intel™ AVX-512 instructions.The server had four gigabytes (GB) of double data rate 4 (DDR4) memorywith an attached one terabyte (TB) NVMe solid state drive (SSD) to storethe input datasets.

Additionally, four datasets were use in the evaluation. The firstdataset was Linux™ source code from version 3 to 4.9, which wasconverted from the tar format to the mtar format that provides betterdeduplication. The other three datasets are images. The size and numberof versions for each dataset are illustrated below in Table 1:

TABLE 1 Name Size (GB) Versions Linux source code 570 1013 Docker Debianimages 20 191 Docker neo4j images 44 130 Docker Cassandra images 26 71

Referring specifically to FIG. 6, a set of graphs illustratingdeduplication ratio improvements of examples of this technology overparallel content-defined chunking (CDC) with variable average chunk sizeand fixed minimum and maximum chunk sizes is illustrated. FIG. 6illustrates the deduplication ratio improvements of this technology overparallel CDC with different average chunk sizes while fixing the minimumand the maximum chunk sizes at two kibibyte (KiB) and 32 KiB,respectively. In particular, the improvements in deduplication ratiousing this technology are 34% for Docker™ neo4j images and 22% forLinux™ source code, at 16 kilobyte (KB) chunks. Additionally, thededuplication ratio improvement increases as the average chunk sizeincreases.

Referring to FIG. 7, a set of graphs illustrating deduplication ratioimprovements of examples of this technology over parallel CDC withvariable minimum chunk sizes is illustrated. FIG. 7 illustrates thatthis technology achieves higher deduplication ratio improvements whenthe minimum chunk size increases. When the minimum chunk size equalshalf of the expected chunk size, the improvement can be as high as 48%for the Docker™ neo4j dataset. When the minimum chunk size becomeslarger, it is more likely that candidate chunk boundaries are locatedwithin the minimum chunk size range, and are thus skipped due to theminimum chunk size constraints. Accordingly, a chunk boundary shift ismore likely to happen leading to lower deduplication for parallel CDC.

Referring to FIG. 8, a set of graphs illustrating improved chunkingspeed of examples of this technology over sequential CDC with variableminimum chunk size is illustrated. In particular, the minimum chunk sizewas varied as it has a significant impact on the chunking performance.As reflected in FIG. 8, with a smaller minimum chunk size, thistechnology provides increased performance, with a peak of 3.3×. When theminimum chunk size increases, the performance increase drops to about2.5×. Additionally, the performance increase of this technology isrelatively stable among all datasets.

With this technology, data deduplication is optimized using a parallelapproach for content-defined variable-size chunking. In particular,content-defined chunking is implemented in two stages: hash computationand comparison and chunk boundary determination. The hash computationand comparison is decoupled from the chunk boundary determination toavoid making chunking decisions without consideration of chunkboundaries from previous segments. The hash computation and comparisonis executed in parallel using threads. Additionally, the chunk boundarydetermination utilizes parallel hardware to analyze a bit array ofpotential chunk boundaries. Leverages the two stages and parallelprocessing, this technology advantageously achieves improved chunkingperformance and deduplication ratio.

Having thus described the basic concept of the invention, it will berather apparent to those skilled in the art that the foregoing detaileddisclosure is intended to be presented by way of example only, and isnot limiting. Various alterations, improvements, and modifications willoccur and are intended to those skilled in the art, though not expresslystated herein. These alterations, improvements, and modifications areintended to be suggested hereby, and are within the spirit and scope ofthe invention. Additionally, the recited order of processing elements orsequences, or the use of numbers, letters, or other designationstherefore, is not intended to limit the claimed processes to any orderexcept as may be specified in the claims. Accordingly, the invention islimited only by the following claims and equivalents thereto.

What is claimed is:
 1. A method, comprising: comparing, by a computingdevice, a hash value to a predefined value for sliding windows inparallel for each of a plurality of segments partitioned from an inputdata stream; parsing, by the computing device, a bit array according tominimum and maximum chunk sizes to identify a plurality of chunkboundaries for the input data stream, wherein the bit array is populatedbased on a result of the comparison and portions of the bit array areparsed in parallel; and storing, by the computing device, unique chunksof the input data stream defined by the chunk boundaries in a storagedevice.
 2. The method of claim 1, wherein each of a plurality of bits ofthe bit array corresponds to one of the sliding windows and one or moreof the sliding windows for a subset of the segments extend into acontiguous one of the segments.
 3. The method of claim 1, furthercomprising setting, by the computing device, one of the bits in the bitarray when the hash value matches the predefined value for one of thesliding windows, wherein the set one of the bits represents a chunkboundary candidate.
 4. The method of claim 3, further comprisingsetting, by the computing device, an n^(th) bit in the bit array whenthe hash value matches the predefined value at an offset of n in theinput data stream.
 5. The method of claim 1, further comprising loading,by the computing device, integers into an Advanced Vector Extensions(AVX) register, wherein each of the integers corresponds to a set ofbits that comprises one of the portions of the bit array and each of thesets of bits follow a last identified one of the chunk boundaries in thebit array by the minimum chunk size.
 6. The method of claim 5, furthercomprising sequentially scanning, by the computing device, the integersin the AVX register to identify one of the integers and left-shiftingthe one of the integers until the left-most bit comprises a valuecorresponding to one of the chunk boundaries, when an AVX instructionindicates that the one of the integers includes the value.
 7. Anon-transitory machine readable medium having stored thereoninstructions for optimized variable-size deduplication using two stagecontent-defined chunking comprising machine executable code which whenexecuted by at least one machine causes the machine to: compare a hashvalue to a predefined value for sliding windows in parallel for each ofa plurality of segments partitioned from an input data stream; parse abit array according to minimum and maximum chunk sizes to identify aplurality of chunk boundaries for the input data stream, wherein the bitarray is populated based on a result of the comparison and portions ofthe bit array are parsed in parallel; and store unique chunks of theinput data stream defined by the chunk boundaries in a storage device.8. The non-transitory machine readable medium of claim 7, wherein eachof a plurality of bits of the bit array corresponds to one of thesliding windows and one or more of the sliding windows for a subset ofthe segments extend into a contiguous one of the segments.
 9. Thenon-transitory machine readable medium of claim 7, wherein the machineexecutable code when executed by the machine further causes the machineto set one of the bits in the bit array when the hash value matches thepredefined value for one of the sliding windows, wherein the set one ofthe bits represents a chunk boundary candidate.
 10. The non-transitorymachine readable medium of claim 9, wherein the machine executable codewhen executed by the machine further causes the machine to set an n^(th)bit in the bit array when the hash value matches the predefined value atan offset of n in the input data stream.
 11. The non-transitory machinereadable medium of claim 7, wherein the machine executable code whenexecuted by the machine further causes the machine to load integers intoan Advanced Vector Extensions (AVX) register, wherein each of theintegers corresponds to a set of bits that comprises one of the portionsof the bit array and each of the sets of bits follow a last identifiedone of the chunk boundaries in the bit array by the minimum chunk size.12. The non-transitory machine readable medium of claim 11, wherein themachine executable code when executed by the machine further causes themachine to sequentially scan the integers in the AVX register toidentify one of the integers and left-shift the one of the integersuntil the left-most bit comprises a value corresponding to one of thechunk boundaries, when an AVX instruction indicates that the one of theintegers includes the value.
 13. A computing device, comprising: amemory containing machine readable medium comprising machine executablecode having stored thereon instructions for optimized variable-sizededuplication using two stage content-defined chunking; and a processorcoupled to the memory, the processor configured to execute the machineexecutable code to cause the processor to: compare a hash value to apredefined value for sliding windows in parallel for each of a pluralityof segments partitioned from an input data stream; parse a bit arrayaccording to minimum and maximum chunk sizes to identify a plurality ofchunk boundaries for the input data stream, wherein the bit array ispopulated based on a result of the comparison and portions of the bitarray are parsed in parallel; and store unique chunks of the input datastream defined by the chunk boundaries in a storage device.
 14. Thecomputing device of claim 13, wherein each of a plurality of bits of thebit array corresponds to one of the sliding windows and one or more ofthe sliding windows for a subset of the segments extend into acontiguous one of the segments.
 15. The computing device of claim 13,wherein the machine executable code to further cause the processor toset one of the bits in the bit array when the hash value matches thepredefined value for one of the sliding windows, wherein the set one ofthe bits represents a chunk boundary candidate.
 16. The computing deviceof claim 15, wherein the machine executable code to further cause theprocessor to set an n^(th) bit in the bit array when the hash valuematches the predefined value at an offset of n in the input data stream.17. The computing device of claim 13, wherein the machine executablecode to further cause the processor to load integers into an AdvancedVector Extensions (AVX) register, wherein each of the integerscorresponds to a set of bits that comprises one of the portions of thebit array and each of the sets of bits follow a last identified one ofthe chunk boundaries in the bit array by the minimum chunk size.
 18. Thecomputing device of claim 17, wherein the machine executable code tofurther cause the processor to sequentially scan the integers in the AVXregister to identify one of the integers and left-shift the one of theintegers until the left-most bit comprises a value corresponding to oneof the chunk boundaries, when an AVX instruction indicates that the oneof the integers includes the value.